System and Method for Assisting a User in Remaining in a Selected Area While the User is in a Virtual Reality Environment

ABSTRACT

A method and apparatus is disclosed for assisting a user, wearing a head mounted display (HMD) that covers a user&#39;s field of vision and has a tracker providing information regarding the position and orientation of the HMD, in locating a physical controller located on a physical base station. A processor causes the HMD to display a virtual world, including a virtual representation of the physical base station and physical controller along with a virtual hand that helps guide the user to the physical base station to allow the user to pick up the physical controller. Another embodiment allows an area in the physical world to be defined within which the user should remain, for example to avoid physical obstacles. The processor causes the HMD to display a warning, such as a virtual fence, to alert the user if the user approaches to within a preselected distance of the boundary.

FIELD OF THE INVENTION

The present invention relates generally to virtual reality devices, andmore particularly to controllers for use in a virtual reality context.

BACKGROUND OF THE INVENTION

Virtual reality is often used to describe a wide variety of applicationscommonly associated with immersive, highly visual, computer-simulatedenvironments that can simulate a user's physical presence in places inthe real world or imagined worlds. While virtual reality can recreate anumber of sensory experiences, the senses most commonly used today tocreate virtual reality appear to be sight and sound.

One method of presenting a virtual world to a user that is commonlyidentified with virtual reality and presently in use is through the useof a visor or helmet containing a video display which encompasses partor all of a user's field of view and presents computer generated imagesrepresenting the virtual reality environment, or “virtual world,” to theuser. Such a device is often referred to as a head-mounted display, orHMD. One type of HMD presently available is the Oculus Rift from OculusVR, now owned by Facebook.

Typically a HMD covers the user's eyes, so that the user sees only thevirtual world while wearing the HMD and is thus unable to see the actualphysical world around the user while in the virtual world. For thisreason, the use of a HMD can create some issues when it is necessary ordesirable that the user interact with, or sometimes not interact with,the real world. Since the user cannot see the real environment, the usermay not know what actions by the user should occur in the real world,and what actions may have detrimental consequences in the real world.

One instance of such an issue may arise when the user wishes to be ableto use some type of input device while in the virtual world. Forexample, a user may wish to use a handheld controller, such as might beused to play a video game, while in a virtual world, and perhaps eventwo controllers, one for each hand. However, since the user cannot seethe physical world once he or she is wearing the HMD, it can bedifficult for the user to locate and pick up the controller afterputting on the HMD, as the user must blindly fumble around the physicalenvironment to locate the controller(s).

Alternatively, it can be difficult to pick up and put on a HMD if theuser has already picked up and is holding a controller in his or herhand, and even more so if there are two controllers. In this case, theuser may drop one or more of the controller(s) or HMD, resulting inpossible damage to the devices.

Another issue with respect to the user's interactions with the physicalworld is that it is difficult for the user to maintain awareness of hisor her location in the physical world while the user sees only thevirtual world. It may be desirable that the user remain within aselected area, for example, if there are physical objects in thevicinity of the user. The user may wish to be able to move within thevirtual world by, for example, taking a step or moving an arm; ofcourse, any such movement also results in the user moving in thephysical world. When playing a conventional video game on a television,for example, even when using a controller which responds to the user'smotion the user is able to see nearby objects, and avoid moving thecontroller in a way which causes the user to collide with such objects.However, when using a HMD the user is unable to see those nearby objectswhile in the virtual world, and may thus not be aware of impendingcollisions.

It is thus desirable to find a solution that will allow a user in avirtual world presented by a HMD to be aware of the user's physicalsurroundings to a sufficient degree to allow the user to pick uphandheld controllers, and to remain in a selected area, for example toavoid collisions with surrounding physical objects.

SUMMARY OF THE INVENTION

A system and method is disclosed which provides visual assistance to auser in a virtual reality environment in locating physical objects whichthe user is unable to see, specifically a physical controller that auser may use to control certain actions or events in the virtual worldand which is located on a physical base station. Another system andmethod is disclosed which provides visual assistance to a user in avirtual reality environment in remaining within a selected area, forexample to avoid physical objects in the user's vicinity which the useris unable to see while in the virtual world.

One embodiment discloses a method of assisting a user wearing ahead-mounted display (HMD) in remaining within a selected area of thephysical world, the user holding a handheld controller having acontroller tracker which provides information regarding a position ofthe handheld controller, the method comprising: receiving, by aprocessor as an input, a definition of a boundary of the selected area;determining, by the processor, the position of the handheld controllerin the physical world using information from the controller tracker;generating and outputting, by the processor, instructions to the HMD todisplay on the HMD a virtual world; determining, by the processor, fromthe position of the handheld controller that the handheld controller iswithin a first preselected distance from the boundary; generating andoutputting, by the processor, instructions to the HMD to display on theHMD a first visual warning in the virtual world.

Another embodiment discloses a system to assist a user in remaining in aselected area of the physical world, the user wearing a head-mounteddisplay (HMD) and holding a handheld controller, comprising: acontroller tracker on the handheld controller which provides informationregarding a position of the handheld controller; and a processorconfigured to: receive as an input a definition of a boundary of theselected area; determine the position of the handheld controller in thephysical world using information from the controller tracker; generateand output instructions to the HMD to display on the HMD a virtualworld; determine from the position of the handheld controller that thehandheld controller is within a first preselected distance from theboundary; and generate and output instructions to the HMD to display onthe HMD a first visual warning in the virtual world.

Still another embodiment discloses a non-transitory computer-readablemedium having embodied thereon a program, the program being executableby a processor to perform a method of assisting a user wearing ahead-mounted display (HMD) in remaining within a selected area of thephysical world, the user holding a handheld controller having acontroller tracker which provides information regarding a position andorientation of the controller, the method comprising the steps of:receiving, by a processor as an input, a definition of a boundary of theselected area; determining, by the processor, the position of thehandheld controller in the physical world using information from thecontroller tracker; generating and outputting, by the processor,instructions to the HMD to display on the HMD a virtual world;determining, by the processor, from the position of the handheldcontroller that the handheld controller is within a first preselecteddistance from the boundary; generating and outputting, by the processor,instructions to the HMD to display on the HMD a first visual warning inthe virtual world.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are illustrations of a user wearing a head mounteddisplay (HMD).

FIG. 2 is an illustration of a generated virtual world including avirtual base station and virtual controllers as it might be seen by auser wearing a HMD.

FIG. 3 is an illustration of a generated virtual world including avirtual base station, virtual controllers and virtual hands as it mightbe seen by a user wearing a HMD.

FIG. 4 is another illustration of a generated virtual world including avirtual base station, virtual controllers and virtual hands as it mightbe seen by a user wearing a HMD as the user approaches a physical basestation.

FIG. 5 is another illustration of a generated virtual world including avirtual base station, virtual controllers and virtual hands as it mightbe seen by a user wearing a HMD after the user has removed a physicalcontroller from the physical base station.

FIG. 6 is another illustration of a generated virtual world includingvirtual hands as it might be seen by a user wearing a HMD after the userhas removed both physical controllers from the physical base station.

FIG. 7 is another illustration of a generated virtual world including avirtual base station, virtual controllers and virtual hands as it mightbe seen by a user wearing a HMD after instructing the processor todisplay the virtual base station and virtual controllers.

FIG. 8 is an illustration of a generated virtual world including avirtual hand holding a virtual sword as it might be seen by a userwearing a HMD after the user has activated a handheld controller in aparticular situation.

FIG. 9 is a flowchart of a method of assisting a user wearing a HMD inlocating a physical handheld controller that is initially placed on aphysical base station according to one embodiment.

FIG. 10 is an illustration of a user and a virtual hand in a virtualworld as the virtual hand might be seen by a user wearing a HMD.

FIG. 11 another illustration of a user, a virtual hand and a visualwarning in a virtual world as virtual hand and visual warning might beseen by a user wearing a HMD the user approaches a boundary defining aselected area in the physical world.

FIG. 12 is an illustration of a selected area in one embodiment.

FIG. 13 is an illustration of a selected area in another embodiment.

FIG. 14 is an illustration of a selected area in still anotherembodiment.

FIG. 15 is a flowchart of a method of assisting a user wearing a HMD inremaining in a selected area according to one embodiment.

DETAILED DESCRIPTION OF THE INVENTION

A method and apparatus is disclosed for assisting a user in locatingphysical objects while the user is wearing a head mounted display (HMD)that covers a user's field of vision and has a tracker allows aprocessor to track the position and orientation of the HMD, the objectsbeing specifically one or more physical controllers that a user may useto control certain actions or events in the virtual world and which arelocated on a physical base station. Another described method andapparatus assists such a user in avoiding physical obstacles in theuser's vicinity. Since the user wearing the HMD sees only a virtualworld, the user is unable to see such physical objects or obstacles.

In one embodiment, when a user activates a display of a virtual worldinto which the user wishes to enter, for example, through the display ofa HMD, a processor tracks the position and orientation of a tracker onthe HMD. The processor also instructs that display to include in thedisplay of the virtual world a virtual representation of the physicalbase station and the physical controller, along with a virtual hand thathelps guide the user to the physical base station so as to allow theuser to pick up the physical controller. Once the physical controllerhas been picked up and the user has moved a certain distance from thephysical base station, a virtual hand may be shown without the virtualcontroller to heighten the sense of realism in the virtual world. Thisallows the user to locate and pick up the physical controller(s) by thesight provided in the virtual world, rather than having to feel aroundto locate the controller(s) by touch.

In another embodiment, a boundary of a selected area is determined, forexample, a “safe area” which the user or another person has determinedis clear of physical obstacles in the physical world so that the usermay move freely without coming into contact with such physicalobstacles. The processor similarly tracks the position and orientationof the HMD tracker, and possibly the position and orientation of one ormore trackers in the physical controllers, and generates instructions tothe HMD to include in a display of a virtual world a visual warning suchas a grid or virtual fence when the user comes within a preselecteddistance of the boundary, thus providing a warning that the user isabout to move out of the selected area.

FIG. 1A shows a user 102 in the real world wearing a HMD 106; HMD 106covers the user's eyes and presents the user with a representation of avirtual world, while preventing the user from seeing the real, orphysical, world.

Attached to or incorporated in the HMD 106 is a tracker 108, which canbe used to determine the position and orientation of the headset 104,and thus of the head of user 102, by, for example, sensing anelectromagnetic field generated from a local base station, video gameconsole or other apparatus (not shown). The tracker 108 providesinformation about the sensed magnetic field to a processor (also notshown), which derives the position and orientation of the tracker 108,and thus HMD 106, relative to a source of the electromagnetic field fromsuch information.

Such trackers, and the processors which determine the position andorientation of such trackers, are well known to those of skill in theart. One such example, U.S. Pat. No. 4,737,794, teaches a “method andapparatus for determining remote object orientation and position with anelectromagnetic coupling.” It shows a plurality of radiating antennaslocated at a source to provide a plurality of electromagnetic fieldsspanning three dimensional space and defining a source referencecoordinate frame, and a plurality of receiving antennas located on anobject to be tracked for receiving that field. A processor receives theoutputs from the receiving antennas and converts the received componentsof the transmitted electromagnetic fields into remote object positionand orientation relative to the source reference coordinate frame. Theantennas may be, for example, dipole antennas, loop antennas, or coilantennas.

Another such example are light-emitting trackers which can be seen by,and therefore provide position and orientation information to, a cameramounted on the base station or elsewhere. Other forms of trackers,including those incorporating inertial sensors (e.g., accelerometersand/or gyroscopes), can likewise be used in the present approach. Aswould be understood by one of ordinary skill in the art in light of theteachings herein, essentially any type of tracker (e.g., magnetic,optical, electrical, sonar, radar, ultrasonic, or any combinationthereof) which can provide information regarding its position (and, insome cases, orientation, if needed or desired) can be used in thepresent approach. Such techniques are all contemplated as being withinthe described system and method, including techniques which can onlyprovide position information, and not orientation, even though the belowdiscussion generally refers to electromagnetic tracking.

FIG. 1A also shows the user 102 holding a controller 110 which the usermay pick up with his or her hand(s), and which allows the user toperform various functions in the virtual world. Various devices suitablefor use as controller 110 will also be known to those of skill in theart. In addition to tracker 108 in HMD 106, handheld controller 110 mayalso have a tracker, from which a processor can derive the position andorientation of handheld controller 110 relative to the base station, byany of the tracking techniques described above. Techniques foridentifying multiple trackers and distinguishing the position andorientation of each are well known to those of skill in the art.

FIG. 1B is a representation of the user 102 of FIG. 1A as the user 102sees the virtual world, showing the field of view 112 that may bepresented to the user by the HMD 106. As will be known to those of skillin the art, a processor (not shown) generates instructions that causeHMD 106 to present a view to user 102 that is appropriate given theposition and orientation of user 102′s head and any actions taken withcontroller 110, so that what is within field of view 112 will change asuser 102 moves his or her head and uses controller 110.

In the illustration of FIG. 1B, the user is seeing the virtual world aspresented on HMD 106. In FIG. 1B the user also sees a virtual object 114in his or her hand, here a virtual gun, rather than the controller 110that the user is actually holding as in FIG. 1A. As will be explainedbelow, the user has presumably used the controller to interact with thevirtual world to “pick up” the virtual gun 114.

In some embodiments, the controller 110 may be placed or “docked” in abase station or video game console (not shown). In some embodiments, thebase station or console may contain the processor as well as the othercomponents needed to determine the position and orientation of thetrackers in the HMD and controllers. For example, the base station maycontain the electromagnetic field source for electromagnetic trackers, acamera for optical detection of certain types of trackers, or otherelements needed to determine the position and orientation of other typesof trackers.

In other embodiments, the base station may be only a cradle or dock uponwhich the controller may rest, perhaps with a groove or contour thatconforms to the shape of the controller, with the processor and/or othercomponents located elsewhere. Alternatively, some components may be inthe base station and some components located elsewhere. If componentssuch as the electromagnetic field source or camera used to determine theposition of the HMD and controller trackers relative to the base stationare not located in the base station, it will be desirable for the basestation to also contain a tracker similar to those used in the HMD andcontrollers, so that the position of the base station is known and thusthe positions of the HMD and controllers relative to the base stationcan be determined.

In the description herein, it is assumed that the position of the HMDand controller trackers relative to the base station may be determinedby any of the tracking means described above, regardless of whether theprocessor and other components needed for determining those relativepositions are located in the base station or elsewhere.

In one embodiment, controller 110 contains a rechargeable battery, andcircuitry in the base station may charge the battery when the controlleris located in the base station. In other embodiments, there may be twocontrollers 110, where it is intended that there is one controller 110for each hand of user 102.

As above, with a HMD 106 which covers the eyes of user 102, once user102 has donned the HMD 106, user 102 can no longer see the surroundingphysical world, including the controller 110 or a physical base stationin which the controller 110 may be placed. User 102 is thus unable todetermine where the controllers are in the physical world in order topick them up other than perhaps by blindly feeling around the area inwhich the user believes the base station might be located.

For this reason, rather than forcing the user to locate the controllerby touch, one embodiment provides a visual representation in the virtualworld of a virtual base station and virtual controller so that the usermay find and pick up the virtual controller by sight in the virtualworld, thus picking up the physical controller without being able to seethe actual physical controller due to the HMD 106 presenting the userwith only a representation of the virtual world.

FIG. 2 shows what a user such as user 102 might see in the virtual worlddisplayed in the HMD 106 in one embodiment. In the illustratedembodiment, the processor has generated instructions to the display ofHMD 106 to create a picture of a virtual world as if the user is lookingtoward a back wall 202, and also displays side walls 204, floor 206 andceiling 208. Also displayed is a virtual base station 210, in which aredocked two virtual controllers 212.

In one embodiment, the processor causes the HMD 106 to display virtualbase station 210 and virtual controllers 212 as having the same generalappearance as the actual physical base station and the physicalcontrollers that the user will pick up and use to control interactionswith the virtual world, and to appear to the user in an apparentlocation in the virtual world which corresponds to the actual locationof the physical base station and controllers in the real world. Thus,although the user is only seeing the virtual world, if the userapproaches the virtual base station 210, he or she will also beapproaching the physical base station in the real world.

As above, in some embodiments a user such as user 102 wears a HMD 106which has attached to it a tracker 108 which can track the position andorientation of the headset 104, and thus the head of user 102 by any ofthe tracking techniques above, such as with respect to anelectromagnetic field generated by a source located in the physical basestation or elsewhere. (Again, the base station should also have atracker if such components as the electromagnetic source or camera usedto track the HMD are not located in the base station.) Thus, as the userapproaches the virtual base station 210 presented on the display, theprocessor receiving information from tracker 108 (and from the basestation tracker where appropriate) can determine that the user is alsoapproaching the physical base station.

FIG. 3 shows what a user might see in the virtual world in oneembodiment as the user has approached the virtual base station 210 inthe virtual world and is looking down towards it, and is alsoapproaching the physical base station in the real world. Now theprocessor has generated instructions to alter the display to the user sothat the perspective has changed, and it appears as if the user iscloser to the virtual base station 210 and looking down towards it. Theback wall 202, side walls 204 and floor 206 are still displayed.

In addition, the instructions from the processor now also cause thedisplay to generate two virtual hands 214 (with a portion of theforearms) reaching toward the virtual base station 210. At this point,since the processor has no information about where the user's hands arein the physical world, the location of these virtual hands 214 isassumed from the position and orientation of the user's head asdetermined from the tracker, and are intended to provide guidance andassistance to the user in locating the physical base station andphysical controllers in the physical world.

As the user gets still closer to the physical base station and physicalcontrollers in the physical world, again as determined from the positionof the tracker of the HMD on the user's head, in one embodiment theprocessor will generate instructions to alter the display as shown inFIG. 4. Now the user appears to be closer to the virtual base station210, and the virtual hands 214 appear to be closer to the virtual basestation 210, than in FIG. 3, although again at this point the locationof virtual hands 214 is assumed from information provided by the trackeron the HMD on the user's head. Again, the virtual base station 210 andvirtual hands 214 assist the user in locating the physical base stationand physical controllers.

FIG. 5 shows what the user might see in the displayed virtual world inone embodiment after grasping one of the physical controllers andremoving it from the physical base station. The processor now generatesinstructions to display the virtual world much as it appears in FIG. 4,with the back wall 202, side walls 204, floor 206, and virtual basestation 210 displayed. However, as will be explained below, now theprocessor is able to determine that the user has removed the rightphysical controller from the physical base station in the physicalworld, and now displays the virtual right hand 214 as holding the rightvirtual controller 212 instead of the right virtual controller 212 beingdocked in the virtual base station as in FIG. 4.

As above, the physical controllers may also have magnetic trackerslocated within them, similar to the tracker in the headset. If so, theprocessor is now able to determine from information received from thetracker in the right physical controller where the user's hand holdingthat controller is located. The processor may then display the virtualright hand 214 in a location within the virtual world that appears tothe user to be where the physical right hand is actually located, ratherthan providing only an estimated location as before.

In one embodiment, the processor is able to detect that the electricalconnection between the right physical controller and the physical basestation has been broken, and thus determine that the right physicalcontroller has been removed from the physical base station. In anotherembodiment, by using information from the tracker in the right physicalcontroller to determine the position of the right physical controllerrelative to the physical base station, the processor is able todetermine that the right physical controller is no longer in thelocation in which it is docked in the physical base station.

At the point in time shown in FIG. 5, since the left physical controllerhas not been removed from the physical base station, the instructionsfrom the processor cause the left virtual controller 212 to still bedisplayed as docked in the virtual base station 210. Similarly, virtualhand 214 is still shown in an estimated location, since the processorstill has no information about where the user's physical left hand isactually located.

Once both controllers have been picked up, to enhance the realism of thevirtual world, it is desirable that the user no longer see the virtualbase station 210 or the virtual controllers 212, i.e., that theinstructions from the processor no longer cause the HMD to show them. Asdiscussed below with respect to FIG. 8, this can, for example, allow theprocessor to cause the HMD to display the user's virtual hands doingsomething in the virtual world, such as grasping a virtual object,rather than holding the virtual controllers (even though the user willof course continue to feel the physical controllers in his or herhands). FIG. 6 shows one embodiment of what might be displayed to a userin a displayed virtual world once both physical controllers have beenpicked up by the user. Instructions from the processor again cause thedisplay of the virtual world to appear much the same as it appears inFIG. 4 or 5, with the back wall 202, side walls 204, and floor 206displayed. However, now the processor has determined that the user hasremoved both physical controllers from the physical base station, it nowdisplays both virtual hands 214 without either the virtual controllers212 or the virtual base station 210 being displayed.

Since the processor is now able to determine from information receivedfrom the controller trackers in the physical controllers where theuser's hands holding the controllers are located, the processor may thendisplay both virtual hands 214 in apparent locations within the virtualworld that correspond to where the user's physical hands are actuallylocated, rather than providing only an estimated location for either orboth virtual hands 214 as before.

At some point, the user may wish to return the physical controllers tothe physical base station. In one embodiment, the user may cause theprocessor to display both the virtual base station 210 and the virtualcontrollers 212, by, for example, entering an appropriate command on acontroller, so that the user may see their locations in the virtualworld. FIG. 7 shows what a user might see once such a point is reached.The instructions from the processor to the HMD again cause the virtualworld to appear much the same as it appears in the previous FIGS. 4 to6, with the back wall 202, side walls 204, and floor 206 displayed. Now,however, the virtual base station 210 and virtual controllers 212 areagain displayed, with the user's two virtual hands 214 holding thevirtual controllers 212, as a user might see his or her hands in thereal world.

As with the virtual hands 214 in FIG. 6, the virtual hands 214 of FIG. 7are shown in apparent locations in the virtual world that correspond tothe actual locations of the user's physical hands in the physical world.Since the processor still has information from the trackers in thephysical controllers being held in the user's hands, the actual locationof the user's hands can still be determined and therefore the virtualhands can be shown by the HMD in such corresponding apparent locations.Since the virtual base station is also in an apparent location in thevirtual world that corresponds to the location of the physical basestation in the physical world, by appearing to place the virtualcontrollers 212 on the virtual base station 210 the user is guided toplace the physical controllers on the physical base station.

In some embodiments, it may be desirable to have the virtual basestation 210 and virtual controllers 212 continue to be displayed untilthe user has moved the physical controllers beyond some shortpreselected distance, for example, a few inches, away from the physicalbase station, and then displayed again when the user has approached towithin the preselected distance. This allows for added realism in thevirtual world, since the user will no longer see the virtual controllersall the time, while maintaining the convenience of seeing the virtualcontrollers near the virtual base station when the physical controllersare close to the physical base station. This may be particularly helpfulin returning the physical controllers to the physical base station, asexplained above. The preselected distance will typically bepre-programmed into the processor as part of the program which generatesand outputs the instructions to the HMD to display the virtual basestation and virtual controllers. Alternatively, the preselected distancemay be selectable by the user in some embodiments.

The processor can continue to generate instructions to the display inthe HMD to present the user with changes in the virtual world based uponactions that the user has performed with one or both of the physicalcontrollers. In a conventional video game, for example, a user may use acontroller to have a character on a television screen “pick up” a weapondisplayed on the screen. Similarly, with the present approach theprocessor can cause a HMD to display a virtual weapon while the user isin a virtual world. Then, upon the user's extending his or her physicalhand(s) in such a way as to cause the corresponding virtual hand(s) to“reach for” the displayed virtual weapon and the user's activating aproper command (e.g., by pressing a button on the controller), theprocessor can cause the HMD to display the user's virtual hand(s) 214 as“picking up” and holding that virtual weapon.

FIG. 8 shows one example of what a user might see in a virtual worldafter the user has extended the user's physical right hand to cause thevirtual right hand to extend toward a virtual sword displayed in thevirtual world on the HMD, and then entered the appropriate command topick up the virtual sword 802. The user's virtual right hand 214 isholding the virtual sword 802, although of course there is no physicalsword and the user's physical right hand is instead still holding thephysical controller. The processor can then display movements of thevirtual sword 802 based upon movements of the user's physical right handas determined from motion of the tracker in the physical controller.

In some embodiments, the processor can provide instructions to the HMDto make the virtual base station 210 always be displayed in the displaywhen the user's head is turned in the direction of the physical basestation. In some alternative embodiments, the processor can provideinstructions to the HMD which result in the virtual base station notbeing displayed when the user is, for example, more than a shortpreselected distance away from the physical base station. (As above, thepreselected distance can be pre-programmed into the processor, or mayalternatively be selectable by the user.) In such cases, as above it maybe desirable to allow the user to perform some action on the physicalcontroller(s) which causes the processor to indicate where the physicalbase station is located, either by displaying the virtual base station210 in the HMD display again, or possibly by providing arrows in the HMDdisplay which indicate the direction the user should look or move toreach the physical base station.

While the above example uses two physical controllers, in someembodiments there may be only a single physical controller. In such acase, the virtual base station 210 and virtual controller 212 may nolonger be displayed in the virtual world when the single physicalcontroller has been removed from the physical base station. Otherchanges to the described method will be apparent to one of skill in theart.

FIG. 9 is a flowchart of a method implemented by a processor that mightresult in the virtual world displays shown in FIG. 2 through 7 aboveaccording to one embodiment. It is assumed that, as above, a user wearsa HMD having a display, a processor provides instructions that directwhat appears on the display, the HMD has a tracker used to track theposition and orientation of the HMD and thus of the user's head, andthere is a base station holding one or more controllers, each alsohaving a tracker. It is also assumed that, if, for example, theelectromagnetic field source or camera used to perform tracking of theHMD and controller trackers is not located in the base station, there isalso a tracker in the base station so that the processor is able todetermine the position and orientation of the base station, and thustrack the position and orientation of the HMD and controller(s) relativeto the base station.

At step 902, the processor generates and outputs instructions to the HMDto display on the HMD display a generated virtual world. At step 904,the processor determines the position and orientation of the HMD fromthe tracker in the HMD, and continues to do so at regular intervals.

At step 906, the processor generates and outputs instructions to the HMDto display on the HMD display a virtual base station, one or morevirtual controllers placed or docked on the virtual base station, and avirtual hand or hands, such as is shown in FIG. 3. As above, at thispoint the virtual hand will generally be in an assumed location basedupon the position and orientation of the tracker in the HMD, and isprovided to help guide the user in locating the physical base stationwith the user's physical hand.

At step 908, the processor determines based upon the position andorientation of the tracker in the HMD (and the tracker in the basestation if appropriate) that the user has moved toward the physical basestation. In response, at step 910 the processor generates and outputsinstructions to the HMD to display on the HMD display the virtual basestation, the virtual controller still placed or docked on the virtualbase station, and the virtual hand, but now with the virtual hand closerto the virtual base station, for example as shown in FIG. 4.

At step 912, the processor determines that the physical controller (ormore than one of them, if there are more than one) has been removed fromthe physical base station, such as is shown in FIGS. 5 and 6. As above,the processor may be able to detect that an electrical connectionbetween the physical base station and the physical controller has beenbroken, such as a connection that allows batteries in the physicalcontroller to be recharged by circuitry in the physical base station.Alternatively, the physical controller may also have a tracker thatallows the processor to determine the position and orientation of thephysical controller, and, once the processor has determined that thedistance between the physical base station and the physical controllerexceeds a certain distance, the processor can therefore determine thatthe physical controller has been removed from the physical base station.

Finally, at step 914 the processor generates and outputs instructions tothe HMD to display on the HMD display the virtual hand, but not thevirtual base station or virtual controller, as shown in FIG. 6. Asabove, the processor can change the instructions so as to no longerdisplay the virtual controller when the physical controller has movedmore than a preselected distance from the physical base station, asdetermined from the position and orientation of the tracker in thephysical controller.

It will be apparent to one of skill in the art that some of these stepsmay be performed in a different order than specified, while othersshould be performed in order to obtain the desired effect. For example,the processor may determine the position and orientation of the HMDtracker before generating and outputting the instructions to the HMD todisplay the generated virtual world. On the other hand, it will mostlikely not be appropriate to send the instructions that result in theHMD no longer displaying the virtual controller before the user hasremoved the physical controller from the physical base station.

As above, the physical controller has a tracker, so the virtual hand maynow be in an apparent location corresponding to the actual location ofthe user's physical hand rather than in an assumed location, since theprocessor will know where the user's physical hand is from the trackerin the physical controller. The ability of the user to see virtual handsin the locations corresponding to the user's physical hands can help theuser to believe in the reality of the virtual world, as well as help inpicking up the physical controllers as described above. As explainedelsewhere herein, this can also help the user to operate in a selectedarea, thereby avoiding physical obstacles in the real world.

In this way, the display of the virtual base station, virtualcontroller(s) and virtual hand(s) allows the user to locate and pick upthe physical controller(s) primarily by sight rather than by touch, eventhough the user is already wearing the HMD and seeing only the virtualworld, and is thus unable to see the physical controller(s). Reversingthe process as explained above similarly allows the user to return thephysical controller(s) to the base station when the user is ready toexit the virtual world, or for other reasons, without having to removethe HMD first.

As above, another issue that may arise when the user is in the virtualworld is that the user may not be able to see the surrounding physicalenvironment and therefore may not be aware of how far the user has movedin the physical world, and may collide with physical obstacles whiletrying to accomplish tasks in the virtual world. For example, if a useris physically located in the user's living room and playing a gamewithin a virtual world, there will likely be physical objects orobstacles present in the room such as a sofa or other furniture, lamps,and even the walls of the room. Since the display in the HMD blocks theuser's view of the physical world, the physical objects are not visibleto the user. Further, since it is generally considered preferable topresent the user with a view of only the virtual world in order to getthe desired immersive effect, these physical items are similarly notpresented to the user through the HMD display (other than the virtualbase station 210 and virtual controllers 212 for the purposes explainedabove). However, if the user cannot tell where the user is, or wherethese physical objects are, while the user in the virtual world, thereis a risk that the user's motions may cause him or her to collide withthese objects, possible resulting in damage to the objects or eveninjury to the user.

Thus, it is desirable to provide a means by which the user may be madeaware that the user is approaching a boundary of a selected area inwhich the user desires to remain, to avoid physical objects or for anyother reason, without distracting the user from the immersive virtualworld. In one embodiment, a selected area having an outer boundary isdetermined, such as a “safe” area in which the user can move freelywithout coming into contact with the physical objects in the user'svicinity. The selected area may already be clear of physical obstacles,or the user or other person may move physical objects to clear theselected area. (Some objects, such as rugs, may not need to be moved.)The processor provides instructions to the HMD display to include adisplay of a virtual fence at the boundary when the user comes within apreselected distance of the boundary of the selected area, thusproviding a warning that the user is approaching the boundary of theselected area.

FIG. 10 shows a representation of a user 1002 in a virtual space. (Asthe display is usually presented from the user's point of view, the user1002 does not see himself or herself, but is shown here for illustrativepurposes only.) As above, the processor instructs the HMD display toshow the user a virtual world, as illustrated here a room including aback wall 1004, side walls 1006, ceiling 1008 and floor 1010. Also asabove, the user will see his or her virtual hand 1012; again, virtualhand 1012 will be in the location of the user's actual hand if the user1002 is holding a controller having a tracker that provides position andorientation information to the processor, and in an assumed location ifthe user 1002 is not holding such a controller. As above, since the useris wearing a HMD that displays the virtual world, the physical world andany physical objects are not visible to the user.

As above the processor is tracking the position and orientation of theuser's head from the tracker of the HMD, and possibly tracking theposition and orientation of the user's hand if the user is holding acontroller with a tracker. Thus, the processor knows where the user'shead and hands are located; if the processor has been provided withinformation about the boundary of the selected area then the processorcan also determine when the user's head and/or hand approaches theboundary.

In one embodiment, when the user's head and/or hand approaches to withina preselected distance, for example, one foot (again, eitherpreprogrammed into the processor or user selectable), of the boundary ofthe selected area the processor generates a warning that the user isclose to the boundary so that the user can take action to avoid theboundary and thus potential collision with the physical objects. In someembodiments, the processor can generate this warning by generatinginstructions to the HMD display to add some virtual element to thevirtual world which can be seen by the user.

FIG. 11 shows one such form of visual warning. In FIG. 11 it is assumedthat the user 1002 is holding a controller with a tracker in the handcorresponding to virtual hand 1012, and that the user 1002 has extendedthat hand to a point within the preselected distance from the boundaryof the selected area. The display in FIG. 11 is similar to that of FIG.10, except that since the user 1002 has extended his hand holding thecontroller, the virtual hand 1012 now extends further in front of theuser 1002.

In addition, since the hand of the user 1002 is within the preselecteddistance of the boundary of the selected area, the processor generatesinstructions to the HMD to generate the visual warning. As illustratedin this embodiment, this visual warning is in the form of a grid oflines or a “virtual fence” located at the boundary, as indicated in FIG.11 by the dashed lines 1114. The grid can be glowing or colored ifdesired to be more visible to the user. In other embodiments, othershapes than a grid-shaped virtual fence can be displayed to the user.For example, the developer of a particular application may choose someother design to provide contrast with, or maintain thematic correctnesswith, the particular virtual world being displayed. In some embodiments,even a uniform translucent glow of a particular color, such as red, maybe appropriate to serve as the visual warning. In addition, thebrightness of the visual warning may increase as the distance the useris from the boundary decreases to provide further notice that the useris getting still closer to the boundary.

The virtual fence 1114 need not cover the entire display of the HMD; itis believed that it will generally be sufficient for the processor togenerate instructions to have the HMD display show the virtual fence1114 in an area around the point in the virtual world at which theuser's head or hand has approached the boundary of the selected area. Tominimize rendering cost, the fence may be rendered as a projectedtexture on small pieces of 3D geometry that cover only the areas wherethe fence is to be visible. A cylindrical projection is calculated byconverting the position of the 3D geometry into cylindrical coordinates,which are used to offset a texture map. This gives the impression thatthe grid lines of the fence are stationary as the 3D geometry is movedto follow the tracker while it is in the proximity of the boundary,minimizing screen coverage of the geometry for optimal processorperformance. Blending is used to create a smooth fade transition on theedges of the geometry, and to make it more transparent as the trackingdevice moves away from the boundary. When the tracker is beyond thepreselected distance, the grid becomes completely transparent and the 3Dgeometry is removed completely to save processing time.

In addition, a different type of visual warning may be appropriate whenthe user is not looking in the direction in which he or she isapproaching the boundary of the selected area. For example, if a user isapproaching the boundary of the selected area but not looking in thedirection of the boundary, if the HMD display is instructed to provideonly a view of the virtual world in front of the user there may be novisual warning. Thus, in these cases, some other type of visual warning,perhaps with directional arrows indicating the direction of the boundaryin relation to the user's view, may be appropriate. It is also possibleto include auditory warnings instead of, or in addition to, a visualwarning.

As above, the fence is displayed in the vicinity of the user's head orhands when the user's head or hands are approaching the boundary asdetermined by the trackers on the user's head or hands. This may requirean earlier warning when it is the user's head that approaches theboundary, since this likely means that the user's body is close to theboundary, and thus that the user's hands may easily go beyond theboundary as a result of an intentional motion or even a reflexivemotion. Thus, the preselected distance within which an approach by theuser causes the processor to display the visual warning may be definedseparately for the HMD tracker and the handheld controller tracker(s).In addition, the preselected distance may vary by application, so that,for example, it is greater in an application in which larger bodymovements are expected from the user than in an application in whichsuch body movements are not expected.

If the user's head or hands go beyond the boundary as determined by thetrackers, the processor can provide additional messaging or warning,such as text or auditory warnings, or an additional visual warning, thatpersists until the user returns to the selected area within theboundary. In addition, if the user's head goes beyond the boundary, theprocessor can cause the HMD to fade to black, and present, for example,only a warning message and/or a beacon or arrows directing the user tomove in the direction of the boundary and the selected area.

The boundary of the selected area may be determined in several ways. Insome embodiments, a person, such as the user or another person, forexample the parent of a child user, may input information into theprocessor indicating that there is a circular area in front of the basestation and having a measured diameter that is clear of objects. FIG. 12illustrates this configuration.

In one embodiment, the information input to the processor is determinedby activating one of the trackers, either in the HMD or a handheldcontroller, at a position 1204 directly in front of a base station 1202.From the position 1204 of the tracker (and the position of a basestation tracker if, for example, the electromagnetic field source orcamera are not in the base station), the processor can determine how faraway the tracker is from the base station 1202, and then determine acircular area 1206 having a radius of that distance centered at thatpoint 1204. In this case, the user should check to see that there are noobstacles within that defined circle since the processor has no way todetermine whether there are any such obstacles in the circular area1206.

In another embodiment, the radius (or diameter) of the circular area1206 is measured (for example, by the user), and the processor receivesas input the measurement. The processor may also receive the fact thatthe circular area is directly in front of the physical base station 1204(again using information from a base station tracker if appropriate), oralternatively the user may be told to set up the physical base station1204 and circular area 1206 in this fashion. In some embodiments, suchinstructions to the user could even include the radius (or diameter) ofa preprogrammed circular area, and direct the user to clear an area ofsuch size of any obstacles.

In some cases, due to the layout of a particular space, or the objectsin the space, it may not be convenient or desirable to locate a desiredcircular selected area immediately in front of the base station as inFIG. 12. Thus, as shown in FIG. 13, in one alternative embodiment, theradius (or diameter) of an open circular area 1306 is measured, e.g., bythe user, and that measurement is received by the processor as input. Atracker is placed at the center of the circular area, and by determiningthe tracker's position and orientation the processor is able todetermine the offset distance and direction to the center of thecircular area from either the base station or another location havingthe electromagnetic field source, camera or other tracking components.This allows the processor to define the circular area having themeasured radius, and centered at the measured offset position, as aselected area.

In still another embodiment, an irregularly shaped area is defined, asshown in FIG. 14. A boundary 1404 around an irregularly shaped area,which is not circular, can be defined, for example, by having the userwalk around a perimeter chosen by the user with one of the trackers todefine the outer boundary of a selected area, for example, one in whichthere are no obstacles. The processor determines the position of thetracker at a number of sampling points as the user moves around thechosen perimeter, and from those points determines the boundary 1404.This approach may be desirable when the size of a circle 1406 that canbe inscribed within boundary 1404 is limited by nearby objects 1408, butthere is some additional room between objects 1408 in which the user canbe free to move without risk of collision with objects 1408.

FIG. 15 is a flowchart of a method implemented by a processor that mightresult in the display of a visual warning such as the virtual fence 1114of FIG. 11, or other warnings as described above. As in the method oflocating objects described above, it is assumed that a user wears a HMDhaving a display, a processor provides instructions that direct whatappears on the display, and the user is holding a handheld controllerhaving a tracker which can be used to track the position and orientationof the handheld controller.

At step 1502, the processor receives as an input information from whichthe processor can define a boundary surrounding a selected area, forexample, one in which there are no physical obstacles. As above, thisinformation will typically be entered by a user or someone else, or mayin part or whole be received from a tracker, and may be the radius andlocation of the center of a circle of a determined diameter that isclear of obstacles, or positions around an irregularly shaped areaprovided by the user or other person walking around the boundary holdinga tracker. Alternatively, as above, the user may be told to clear anyobstacles from a circular area of a preprogrammed radius or diameter.

At step 1504, the processor determines the position of the handheldcontroller from the controller tracker. Next, at step 1506, theprocessor generates and outputs instructions to the HMD to display onthe HMD display a generated virtual world.

Next, at step 1508, the processor determines from the position of thehandheld controller that the handheld controller is within a preselecteddistance of the boundary. Finally, at step 1510 the processor generatesand outputs instructions to the HMD to display on the HMD a visualwarning to the user. As above, this visual warning can be in the form ofgrid lines such as a virtual fence, or some other configuration. Inappropriate situations an audible warning can be generated in additionto, or in lieu of, the visual warning.

As with the method of assisting a user in locating a physical controllerabove, a person of skill in the art will appreciate that some of thesteps of the method of assisting a user in avoiding physical objects maybe performed in a different order, while certain steps will be mostadvantageously performed in a particular order.

Also as above, the user is wearing a HMD that may also have a tracker.In such a case, the processor may also display a visual warning when theuser's head approaches the boundary. Since the user does not see his orher own head in the virtual world, the visual warning in this case may,for example, again be in the form of grid lines, and appear in thevirtual world to be immediately in front of the user.

Also as above, in various embodiments, the processor can generate thevisual warning in the form of grid lines or a virtual fence, or in anyother desired shape or configuration. In some embodiments, it may againbe appropriate to generate an audio warning as well, or instead of, thevisual warning.

By alerting a user in this fashion that the user is approaching theboundary of the selected area, the described system and method allows auser to take action to avoid the boundary and stay within the selectedarea defined by that boundary. This can assist the user in avoidingcollision with physical objects in the area of the user, and anypossible resulting damage to such objects or injury to the user.

The disclosed system and method has been explained above with referenceto several embodiments. Other embodiments will be apparent to thoseskilled in the art in light of this disclosure. Certain aspects of thedescribed method and apparatus may readily be implemented usingconfigurations or steps other than those described in the embodimentsabove, or in conjunction with elements other than or in addition tothose described above. It will also be apparent that in some instancesthe order of steps described herein may be altered without changing theresult of performance of all of the described steps.

For example, it is expected that the described apparatus may beimplemented in numerous ways. As noted above, the controller(s) may nestor dock in a base station, or may be separate handheld controllersfamiliar to users of video game consoles. As above, various embodimentsmay be applicable to HMDs that block the user's view of all or only partof the physical world. Even some uses of a device such as Google Glass,which obscures only a small portion of the user's field of vision of thephysical world, may benefit from some embodiments described herein.

There may be a single processor, or multiple processors performingdifferent functions of the functions described herein. As above, aprocessor may be located in the base station, or in a separate location,or even in the HMD or one of the controllers if desired. One of skill inthe art will appreciate how to determine which and how many processorswill be appropriate for a specific intended application. Similarly, asexplained above, various tracking techniques may be used in addition toelectromagnetic tracking, and in some embodiments some of the othercomponents used for tracking, such as an electromagnetic field source ora camera, may be located in the base station, while in other embodimentsthey may be located elsewhere.

One of skill in the art will also appreciate that controllers may be ofvarious shapes other than those shown or described herein. It may bedesirable to have a controller in the shape of an object that isintended to appear in the virtual world so that the user's hands “feel”the same object that is seen in the HMD; for example, a controller maytake the shape of a bat, tennis racket, golf club or other sportingimplement (or the handle of such an object), a gun, sword or otherweapon (or again the handle thereof), or some other toy or device whichwill appear in the virtual world.

These and other variations upon the embodiments are intended to becovered by the present disclosure, which is limited only by the appendedclaims.

What is claimed is:
 1. A method of assisting a user wearing ahead-mounted display (HMD) in remaining within a selected area of thephysical world, the user holding a handheld controller having acontroller tracker which provides information regarding a position ofthe handheld controller, the method comprising: receiving, by aprocessor as an input, a definition of a boundary of the selected area;determining, by the processor, the position of the handheld controllerin the physical world using information from the controller tracker;generating and outputting, by the processor, instructions to the HMD todisplay on the HMD a virtual world; determining, by the processor, fromthe position of the handheld controller that the handheld controller iswithin a first preselected distance from the boundary; generating andoutputting, by the processor, instructions to the HMD to display on theHMD a first visual warning in the virtual world.
 2. The method of claim1 wherein: generating and outputting instructions to the HMD to displaythe virtual world further comprises generating and outputtinginstructions to the HMD to display on the HMD a virtual hand in thevirtual world, the virtual hand having an apparent location in thevirtual world that corresponds to the position of the handheldcontroller in the physical world; and the first visual warning is apattern of grid lines in the vicinity of the displayed virtual hand. 3.The method of claim 1, wherein the HMD has a HMD tracker which providesinformation regarding the position and orientation of the HMD, themethod further comprising: determining, by the processor, the positionof the HMD in the physical world using information from the HMD tracker;determining, by the processor, from the position of the HMD that the HMDis within a second preselected distance of the boundary; generating andoutputting, by the processor, instructions to the HMD to display on theHMD a second visual warning in the virtual world.
 4. The method of claim4 wherein the second visual warning is a pattern of grid lines having anapparent location in the virtual world immediately in front of the user.5. The method of claim 3 wherein the first visual warning and the secondvisual warning are the same.
 6. The method of claim 3 wherein the firstvisual warning and the second visual warning are different.
 7. Themethod of claim 3 wherein the first preselected distance and the secondpreselected distance are the same distance.
 8. The method of claim 3wherein the first preselected distance and the second preselecteddistance are different distances.
 9. The method of claim 1 whereinreceiving a definition of a boundary of the selected area furthercomprises: determining in sequence, by the processor, the location ofthe handheld controller in the physical world at each of a plurality ofpoints using information from the controller tracker; and connecting insequence, by the processor, the plurality of determined locations toform the boundary.
 10. The method of claim 1 wherein receiving adefinition of a boundary of the selected area further comprises:determining, by the processor, a location of the handheld controller inthe physical world using information from the controller tracker;receiving, by the processor, a distance measurement; and defining, bythe processor, the boundary as having a circular shape, centered at thedetermined location and having a radius equal to the received distancemeasurement.
 11. A system to assist a user in remaining in a selectedarea of the physical world, the user wearing a head-mounted display(HMD) and holding a handheld controller, comprising: a controllertracker on the handheld controller which provides information regardinga position of the handheld controller; and a processor configured to:receive as an input a definition of a boundary of the selected area;determine the position of the handheld controller in the physical worldusing information from the controller tracker; generate and outputinstructions to the HMD to display on the HMD a virtual world; determinefrom the position of the handheld controller that the handheldcontroller is within a first preselected distance from the boundary; andgenerate and output instructions to the HMD to display on the HMD afirst visual warning in the virtual world.
 12. The system of claim 11wherein the processor is further configured to generate and outputinstructions to the HMD to display on the HMD a virtual hand in thevirtual world, the virtual hand having an apparent location in thevirtual world that corresponds to the position of the handheldcontroller in the physical world; and wherein the first visual warningis a pattern of grid lines in the vicinity of the displayed virtualhand.
 13. The system of claim 11 further comprising a HMD tracker on theHMD which provides information regarding the position and orientation ofthe HMD, and wherein the processor is further configured to: determinethe position of the HMD in the physical world using information from theHMD tracker; determine from the position of the HMD that the HMD iswithin a second preselected distance of the boundary; generate andoutput instructions to the HMD to display on the HMD the visual warning.14. The system of claim 13 wherein the second visual warning is apattern of grid lines having an apparent location in the virtual worldimmediately in front of the user.
 15. The system of claim 13 wherein thefirst visual warning and the second visual warning are the same.
 16. Thesystem of claim 13 wherein the first visual warning and the secondvisual warning are different.
 17. The system of claim 13 wherein thefirst preselected distance and the second preselected distance are thesame distance.
 18. The system of claim 13 wherein the first preselecteddistance and the second preselected distance are different distances.19. The system of claim 1 wherein to receive as an input the definitionof a boundary the processor is further configured to: determine insequence the location of the handheld controller in the physical worldat each of a plurality of points using information from the controllertracker; and connect in sequence the plurality of determined locationsto form the boundary.
 20. The system of claim 1 wherein to receive as aninput the definition of a boundary the processor is further configuredto: determine a location of the handheld controller in the physicalworld using information from the controller tracker; receive a distancemeasurement; and define the boundary as having a circular shape,centered at the determined location and having a radius equal to thereceived distance measurement.
 21. A non-transitory computer-readablemedium having embodied thereon a program, the program being executableby a processor to perform a method of assisting a user wearing ahead-mounted display (HMD) in remaining within a selected area of thephysical world, the user holding a handheld controller having acontroller tracker which provides information regarding a position andorientation of the controller, the method comprising the steps of:receiving, by a processor as an input, a definition of a boundary of theselected area; determining, by the processor, the position of thehandheld controller in the physical world using information from thecontroller tracker; generating and outputting, by the processor,instructions to the HMD to display on the HMD a virtual world;determining, by the processor, from the position of the handheldcontroller that the handheld controller is within a first preselecteddistance from the boundary; generating and outputting, by the processor,instructions to the HMD to display on the HMD a first visual warning inthe virtual world.